Sample Efficient Reinforcement Learning with REINFORCE

Policy gradient methods are among the most effective for large-scale reinforcement learning, and their empirical success has prompted several works that develop foundation of global convergence theory. However, prior have either required exact gradients or state-action visitation measure based mini-batch stochastic with a diverging batch size, which limit applicability in practical scenarios. In this paper, we consider classical policy compute an approximate single trajectory fixed size trajectories under soft-max parametrization log-barrier regularization, along widely-used REINFORCE estimation procedure. By controlling number "bad" episodes resorting to doubling trick, establish anytime sub-linear high probability regret bound as well almost sure average asymptotically rate. These provide first set sample efficiency results well-known algorithm contribute better understanding its performance practice.